A Dynamical System Approach to Intrusion Detection Using System Call Analysis

1 This research work was funded, in part, by an MRI grant from the National Science Foundation (Grant #: CNS – 0619069). ABSTRACT Code injections can aid successful intrusion attempts, thereby allowing viruses and worms to spread. Current research into intrusion detection is notably focused on application behavior profiling through system call trace analysis. Studying the system call layer has been identified as a potential approach to render revealing details about an application’s behavior. System call sequences available from the execution trace of an application can be subjected to different modeling techniques to approximate the application’s normal execution. This research views application programs as dynamical systems, and applies dynamical system analysis tools operating on time series data, merely the system calls made by an application, to identify the degree of determinism in a dynamical system. There is some prior work in the literature analyzing programs as dynamical systems, but they lack proper utilization of dynamical system formalisms and associated analysis tools. In our research we utilize a set of dynamical system analysis tools composed of Approximate Entropy, Central Tendency Measure, and Recurrence Plot derived measures. Our initial results are promising in detecting code injections.